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Why GAO Did This Study 

Search and rescue — one of the 
United States Coast Guard's 
(USCG) oldest missions and 
highest priorities — involves 
minimizing loss of life, injury, and 
property damage by aiding people 
and boats in distress. In September 
2002, USCG contracted to replace 
its search and rescue 
communications system — installed 
in the 1970s — with a new system 
known as Rescue 21. However, the 
acquisition and initial 
implementation of Rescue 21 has 
resulted in significant cost 
overruns and schedule delays. 

GAO's objectives in reviewing the 
Rescue 21 program were to (1) 
assess the reasons for the 
significant cost overruns and 
implementation delays; (2) evaluate 
the viability of the revised cost and 
schedule estimates; and (3) 
evaluate the impact of the 
implementation delays. 



What GAO Recommends 



GAO recommends that the 
Commandant of USCG ensure that 
executive-level management 
oversees Rescue 21's progress 
toward cost and schedule 
milestones and manages risks; 
establishes milestones to complete 
an integrated baseline review; and 
develops revised cost and schedule 
estimates. The Department of 
Homeland Security agreed with the 
recommendations and has begun to 
implement them; however, the 
department expressed concerns 
with selected aspects of the report. 



What GAO Found 

Key factors that contributed to Rescue 21 cost overruns and schedule delays 
were inadequacies in requirements management, project monitoring, risk 
management, contractor cost and schedule estimation and delivery, and 
executive-level oversight (see table). Accordingly, the estimated total 
acquisition cost for Rescue 21 has increased from $250 million in 1999 to 
$710.5 million in 2005, and the timeline for achieving full operating capability 
has been delayed from 2006 until 2011. USCG officials agreed that 
improvements need to be made to the management of the Rescue 21 project, 
and they are taking steps to address some of these issues. 

USCG's current acquisition cost estimate of $710.5 million is not viable. Our 
analysis of contractor performance trends indicates that additional overruns 
will likely bring the total acquisition cost to $872 million, unless critical 
changes are made. Additionally, USCG's schedule is uncertain due to on- 
going contract item renegotiations and pending decisions regarding vessel 
asset tracking functionality. Finally, further cost increases may result from 
the variability of costs for tower preparation and construction. To improve 
its current cost and schedule estimates, USCG plans to complete an 
integrated baseline review after contract items are renegotiated; however, a 
date for completing this review has not been established. 

The delays in deploying Rescue 21 could affect sites awaiting modernization. 
For example, key functionality, such as improved direction finding and 
improved coverage, will not be available for a significant period of time, and 
legacy equipment may fail, requiring costly repairs and upgrades to address 
coverage gaps and other operational issues. 

Key Factors in Rescue 21 Cost Overruns and Schedule Delays 

Requirements 
management 

Project monitoring 



USCG did not follow a rigorous requirements management process 
and testing revealed incomplete and poorly defined requirements. 

USCG did not effectively use earned value management data to 
measure performance and take corrective action on negative trends. 

USCG did not always effectively mitigate and communicate risks. 

The contractor created a schedule that underestimated the time 
required to complete key tasks, and development took longer than 
planned, which led to delays in testing. 

USCG stated that it had an executive-level oversight process that 
included semiannual and key decision point reviews. However, there 
is no evidence that these reviews of Rescue 21 occurred before 2005. 

Source: GAO. 



Risk management 

Contractor cost and 
schedule estimation and 
delivery 

Executive-level oversight 
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United States Government Accountability Office 
Washington, DC 20548 



May 31, 2006 

The Honorable Harold Rogers 
Chairman 

The Honorable Martin Sabo 
Ranking Member 

Subcommittee on Homeland Security, 
Committee on Appropriations 
House of Representatives 

The Honorable Olympia Snowe 
Chairman 

The Honorable Maria Cantwell 
Ranking Member 

Subcommittee on Fisheries and Coast Guard, 
Committee on Commerce, Science and Transportation 
United States Senate 

The United States Coast Guard (USCG) is currently replacing its 30-year- 
old search and rescue communications systems with a new system known 
as Rescue 21. Rescue 21 will be used not only for search and rescue, but it 
will support other USCG missions, including those involving homeland 
security. However, Rescue 21 has had significant cost overruns and 
schedule delays. As a result of these delays, completion of Rescue 21 
deployment nationwide — originally scheduled for fiscal year 2006 — has 
been delayed until fiscal year 2011, and its total acquisition cost has risen 
from $250 million in 1999 to its current estimate of $710.5 million. 1 In 
addition to the cost overruns and schedule delays, Rescue 21 may not be 
deployed with all of the functionality originally promised. For example, 
functionality for vessel asset tracking was not deployed as part of the 
initial operating capability due to the system's inability to meet original 
requirements, and USCG is now considering alternative strategies for this 
functionality. Also, there has been a reduction in promised improvements 
to limit communications gaps; originally, Rescue 21 was intended to limit 
communications gaps to 2 percent — now the target is less than 10 percent. 



April 2006, the Department of Homeland Security approved a new acquisition program 
baseline for Rescue 21 with a total acquisition cost of $730.2 million after decreasing 
certain functionality. 
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This report responds to your request that we (1) assess the reasons for the 
significant implementation delay and cost overruns against the original 
Rescue 21 proposal, (2) evaluate the viability of the USCG's revised cost 
and implementation schedule, and (3) evaluate the impact of Rescue 21's 
implementation delay on USCG's field units that are awaiting the 
modernization of antiquated communications equipment. 

On March 27, 2006, we provided your staff with a briefing on the results of 
our study. The slides from that briefing are included in this report as 
appendix I. The purpose of this report is to formally publish the briefing 
slides and officially transmit our recommendations to the Commandant of 
USCG. 

To assess the reasons for the implementation delay and cost overruns of 
Rescue 21, we analyzed current and previous program documents, 
including current and previous acquisition plans, cost estimates, status 
reports, schedules, promised functionality, earned value management 
(EVM) 2 data, and cost analyses. We also interviewed key officials and 
contractors. To evaluate the viability of the USCG's revised cost estimate 
and implementation schedule, we analyzed program cost and schedule 
documents, cost estimating methodologies, and changes to development 
and deployment plans. To identify factors that could affect the program 
baseline in the future, we assessed the prime contractor's performance 
related to cost and schedule. We applied established earned value analysis 
techniques to data from contractor cost performance reports that spanned 
a 26-month period — from October 2003 to December 2005 to show 
trends in cost and schedule performance and to estimate the likely costs at 
the completion of the prime contract. To evaluate the impact of Rescue 
21's implementation delay, we reviewed reports on legacy system failures 
and anticipated failures and interviewed field officials responsible for 
overseeing legacy system operations at 11 field units, the Maintenance and 
Logistic Commands for the Pacific and Atlantic Areas, and Rescue 21 users 
at the two initial operating regions. 



2 EVM is a project management tool that integrates the investment scope of work with 
schedule and cost elements for investment planning and control. This method compares 
the value of work accomplished during a given period with that of the work expected in the 
period. Differences in expectations are measured in both cost and schedule variances. The 
Office of Management and Budget (OMB) requires agencies to use EVM as part of their 
performance-based management system for any investment under development or with 
system improvements under way. 
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We performed our work at USCG headquarters and at selected field units 
and communications centers from December 2005 through March 2006, in 
accordance with generally accepted auditing standards. 

In summary, we made the following major points in our briefing: 

• Key factors that contributed to Rescue 21 cost overruns and schedule 
delays were shortcomings in requirements management, project 
monitoring, risk management, contractor cost and schedule estimation 
and delivery, and executive-level oversight. Among the points we made 
were that USCG did not effectively use EVM data to measure performance 
and take corrective actions; USCG, while identifying, analyzing, and 
prioritizing risks, did not always mitigate and communicate risks; and 
USCG stated that it had an executive oversight process that included 
semiannual and key decision point reviews. However, there is no evidence 
that these reviews of Rescue 21 occurred before 2005. USCG agreed that 
improvements need to be made in the management of the Rescue 21 
project, and it is taking steps to address some of these issues. 

• The USCG's current acquisition cost estimate of $710.5 million is not 
viable: based on our analysis of contractor performance trends and the 
renegotiation of 91 contract items for site deployment, we forecast a cost 
overrun of approximately $161 million, which will likely bring total 
acquisition costs to $872 million, unless critical changes are made. Also, 
the USCG's schedule is uncertain because of the continuing contract item 
renegotiations and decisions regarding vessel asset tracking functionality. 
Finally, these additional factors may increase costs and delay schedules: 
(1) the variability of costs for tower preparation and construction and (2) 
site reorganization. 3 USCG realizes that it needs to improve its cost and 
schedule management and has plans to complete an integrated baseline 
review for the contract items as they are renegotiated; however, a date for 
completing this review has not been established. 

• The delays in deploying Rescue 21 could have several effects: (1) key 
functionality, such as improved direction finding and improved coverage, 
will not be available for a significant period of time; (2) legacy equipment 
may fail, requiring costly repairs; and (3) legacy equipment may require 
upgrades to address coverage gaps and other operational issues. For 
example, USCG instituted a moratorium on upgrades to the legacy system 



3 USCG is currently reorganizing its field units into new, integrated, and standardized 
organizations referred to as sectors. 
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because Rescue 21 was to replace the equipment. However, despite the 
moratorium, according to USCG officials, some field units have upgraded 
their legacy systems to address operational challenges. Upgrades such as 
these would result in higher costs, especially at field units scheduled for 
implementation at a later date. 



Recommendations for ^° more effectively manage the remaining development and deployment 

of Rescue 21, we recommend that the Commandant of USCG ensure that 

Executive Action USCG executive-level management implements the following three 

recommendations: 

• Oversee the project's progress toward cost and schedule milestones and 
management of risks. 

• Establish a milestone to complete Rescue 21's integrated baseline review, 
to include all renegotiated contract items. 

• Use the results of this baseline review to complete a revised cost and 
schedule estimate. 



In providing written comments on a draft to this report, the Department of 
Homeland Security's (DHS) Director of the Departmental GAO/Office of 
the Inspector General Liaison Office concurred with our recommendations 
and stated that USCG would continue to aggressively pursue the 
recommendations in the GAO report in order to ensure that Rescue 21 
provides critical life-saving capabilities and that the acquisition is 
effectively managed. DHS stated that it was currently addressing our 
recommendations and provided examples of efforts to date. Specifically, 
DHS and USCG increased oversight with quarterly flag officer/vice 
president-level program reviews and periodic DHS Joint Requirements 
Council and Investment Review Board reviews in October 2005 and plan to 
initiate monthly progress reviews of the project schedule beginning in 
June 2006. 

Regarding the cost of Rescue 21, DHS stated that it had recently revised its 
acquisition program baseline. The revised total acquisition cost for Rescue 
21 is $730.2 million, which was achieved by adjusting several performance 
parameters and terminating the contractor's option to provide vessel 
subsystem components. DHS plans to achieve this capability through other 
means. However, we continue to stress the importance of our 
recommendation that calls for the completion of an integrated baseline 



Agency Comments 
and Our Evaluation 
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review. If performed correctly, an integrated baseline review will provide 
insight into the risks associated with the $730.2 million estimate and the 
extent to which the estimate is achievable. 

DHS agreed that USCG has responsibilities for managing schedule delays 
and cost overruns in major acquisition projects and with our findings 
regarding the contractor's inability to manage cost and schedule. DHS 
stated that the root cause for the systems deployment delays and costs 
overruns was poor estimates and failed performance by the contractor. 
The department, however, raised concerns with four of our findings 
regarding USCG management and oversight and provided us with some 
additional information regarding these findings that did not fully address 
our concerns. Therefore, we continue to believe that shortcomings in 
USCG's management and oversight of Rescue 21 contributed to cost 
overruns and schedule delays. USGC concerns and our response 
associated with each of the four areas follows. 

First, DHS stated that our findings on requirements management are not 
consistent with our previously reported 2003 findings that USCG did have 
a process for managing requirements and it cited various requirements 
management documents. Our current finding states that USCG did not 
follow rigorous requirements management processes for Rescue 21 — not 
that the USCG did not have a process — and is therefore consistent with 
our previous report. DHS also stated that testing revealed that 
requirements were clearly defined. However, as stated in our briefing, a 
USCG usability and operability assessment of Rescue 21 stated that most 
of the operational advancements envisioned for the project were not 
achieved, and these problems could have been avoided by including user 
requirements in the contract. DHS's response also cites software and 
hardware defects that can be attributed to shortcomings in requirements 
management, providing further support for our finding that USCG did not 
follow a rigorous requirements management process. 

Second, DHS disagreed with our finding that they did not effectively use 
EVM data to measure performance and take corrective actions. Evidence 
showing USCG actions to address schedule delays from EVM data was not 
provided to us during our review. On May 8, 2006, 6 weeks after the 
briefing on March 27, 2006, DHS provided documentation showing that 
some actions were taken by USCG in 2004. However, this is nearly a year 
after the EVM data began showing schedule slippages in March 2003 and 
after the Rescue 21 project missed its initial operating capability date of 
September 2003. Further, DHS stated that the contractor undertook a 
significant reorganization of its management structure in response to its 
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concerns related to cost and schedule performance. Because this 
information was not mentioned to us during our discussions with either 
USCG or the contractor during our review, on May 15, 2006, we contacted 
both the contractor's Rescue 21 project manager and contract manager to 
clarify the reorganization statement. These individuals told us that changes 
in contracting personnel were due to internal movement between ongoing 
contracts and did not constitute a reorganization. 

Third, DHS disagreed with our finding that risks were not always mitigated 
and communicated. DHS stated that there were various reports and 
meetings where risks were reported and discussed. Documentation for 
some of these reports and meetings was provided to us on May 8, 2006. 
Furthermore, as stated in the briefing, there were examples of risks that 
were not effectively mitigated or communicated. The Human System 
Interface (HSI) work group referred to in the agency's response was 
established only in 2005, after Rescue 21 had shown significant HSI issues 
in its final operational testing and evaluation phase. Effective risk 
management would have identified and addressed risks earlier in the 
system life cycle. 

Finally, DHS disagreed with our finding that there was no evidence to 
support its statement that there was executive-level management oversight 
prior to 2005. During our review, DHS cited a list of semiannual and key 
decision point reviews, dating from 2002 to 2005; however, as stated in our 
briefing, it did not provide evidence of the pre-2005 briefings. On May 8, 
2006, DHS provided some evidence that these meetings occurred; 
however, based on our review of the documentation, it was unclear which 
executives attended these meetings and, therefore, whether the 
appropriate executives were in attendance. Furthermore, the department 
did not provide evidence showing executive direction to mitigate risks or 
address problems. Finally, the department stated that it had increased 
oversight in October 2005 by establishing quarterly flag officer/vice 
president-level program reviews and periodic DHS Joint Requirements 
Council and Investment Review Board reviews. This additional oversight 
provides further support for our finding on the lack of executive level 
oversight prior to 2005. DHS also challenged our finding that executive- 
level monitoring of the risk mitigation activities that resulted from the 
2005 meetings did not occur, citing various monitoring activities that 
occurred. However, it did not provide evidence of the monitoring 
activities. 
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DHS also provided technical comments that were incorporated in this 
report where appropriate. DHS's written comments are reprinted in 
appendix II. 



We are sending copies of this report to the Secretary of Homeland 
Security, the Commandant of the United States Coast Guard, and other 
interested parties. We will make copies available to others on request. In 
addition, the report will be available at no charge on the GAO Web site at 
http://www.gao.gov. 

If you or your staff have any questions concerning this report, please 
contact David A. Powner at (202) 512-9286 or by e-mail at 
pownerd@gao.gov or Stephen L. Caldwell at (202) 512-9610 or by e-mail at 
caldwells@gao.gov. 



David A. Powner 

Director, Information Technology Management Issues 






Stephen L. Caldwell 

Acting Director, Homeland Security and Justice Issue 



Page 7 



GAO-06-623 Coast Guard's Rescue System 



Appendix I: Briefing Slides 



k GAP 

Accountability * Integrity * Reliability 

Management and Oversight Improvements 
Needed for Coast Guard's Rescue 21 Acquisition 



Briefing for 

the House Committee on Appropriations, Subcommittee on 
Homeland Security 

and 

the Senate Committee on Commerce, Science and Transportation, 
Subcommittee on Fisheries and Coast Guard 

March 27, 2006 
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Briefing Overview 


Introduction 




Objectives, Scope, and Methodology 




Results in Brief 




Background 




Objective 1 : Reasons for Cost Overruns and Schedule Delays 




Objective 2: Revised Cost and Schedule Estimates 




Objective 3: Delay in Implementation 




Conclusions 




Recommendations 




Agency Comments 
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p. vJ jHl \J Introduction 



Accountability * Integrity * Reliability 



Search and rescue is one of the United States Coast Guard's (USCG) oldest missions and 
highest priorities. The search and rescue mission includes minimizing the loss of life, injury, 
and property damage by aiding people and boats in distress. 



The National Distress and Response System (NDRS) is the legacy communications 
component of USCG's search and rescue program. However, the 30-year-old system has 
several deficiencies and is difficult to maintain, according to USCG officials. In September 
2002, the USCG contracted with General Dynamics Decision Systems (General Dynamics) 
to modernize and replace the NDRS with a system called Rescue 21. 

In October 2005, due to significant cost increases and schedule delays associated with the 
Rescue 21 program, the Chairman and Ranking Member of the House Appropriations 
Subcommittee on Homeland Security requested that we evaluate the Rescue 21 program. 
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^ Accoun,a bili ,vMn,e gfity ^e ll a bili t y Objectives, Scope, and Methodology 

As agreed, our objectives were to 

■ assess the reasons for the significant implementation delay and cost overruns against 
the original Rescue 21 proposal; 

■ evaluate the viability of the USCG's revised cost and implementation schedule; and 

■ evaluate the impact of Rescue 21 's implementation delay on the USCG's field units that 
are awaiting the modernization of antiquated communications equipment. 



To assess the reasons for the implementation delay and cost overruns of Rescue 21 , we 
analyzed Rescue 21 documents, including current and previous acquisition plans, cost 
estimates, status reports, schedules, promised functionality, earned value management 
(EVM) 1 data, and cost analyses. We also interviewed key officials and contractors involved 
in the development and deployment of Rescue 21 . 



1 EVM is a project management tool that integrates the investment scope of work with schedule and cost elements for investment planning and control. This method compares the value 
of work accomplished during a given period with that of the work expected in the period. Differences in expectations are measured in both cost and schedule variances. OMB requires 
agencies to use EVM as part of their performance-based management system for any investment under development or with system improvements under way. 
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Objectives, Scope, and Methodology 

To evaluate the viability of USCG's revised cost estimate and implementation schedule, we 
analyzed current program cost and schedule estimates, cost estimating methodologies, 
and changes to development and deployment plans. To identify factors that could affect the 
program baseline in the future, we assessed the prime contractor's performance related to 
cost and schedule. We applied established earned value analysis techniques to data from 
contractor cost performance reports that spanned a 26-month period, from October 2003 to 
December 2005, to show trends in cost and schedule performance and to estimate the 
likely costs at the completion of the prime contract. 

To evaluate the impact of Rescue 21 's implementation delay, we reviewed reports on 
legacy system failures and anticipated failures as well as documentation from USCG sites 
regarding problems associated with the system. We also interviewed field officials 
responsible for overseeing legacy system operations at 1 1 field units. We visited five of 
these field units (Humboldt, California; San Francisco, California; New Orleans, Louisiana; 
Mobile, Alabama; and Miami, Florida) and interviewed officials by telephone at six other 
locations (Honolulu, Hawaii; Guam, Territory of Guam; Detroit, Michigan; Valdez, Alaska; 
Kodiak, Alaska; and Juneau, Alaska). These locations were selected based on coverage 
gaps previously identified and their significance to search and rescue and homeland 
security operations. 
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Accountability * Integrity * Reliability 

Objectives, Scope, and Methodology 

We also interviewed Maintenance and Logistic Commands for the Pacific and Atlantic 
Areas and Rescue 21 users at the initial operating sites at the Atlantic City and Eastern 
Shore communications centers. We compared planned Rescue 21 functionality with what 
was actually delivered. 

We performed our work at USCG headquarters and selected field units and 
communications centers from December 2005 through March 2006, in accordance with 
generally accepted auditing standards. 



6 



Page 13 



GAO-06-623 Coast Guard's Rescue System 



Appendix I: Briefing Slides 



4 C A O 

p. VJ r\ \J Results in Brief 



Accountability * Integrity * Reliability 



Key factors that contributed to Rescue 21 cost overruns and schedule delays were 
shortcomings in requirements management, project monitoring, risk management, 
contractor cost and schedule estimation and delivery, and executive-level oversight. 
USCG agreed that improvements need to be made in these areas and is taking steps to 
address these issues. 



The Coast Guard's current acquisition cost estimate of $710.5 million is not viable: our 
analysis of contractor performance trends indicates that additional cost overruns will 
likely bring total acquisition costs to $872 million unless critical changes are made. This 
includes a significant number of contract items that have not been completed as planned 
and must now be renegotiated. Also, USCG's schedule, which now shows full operating 
capability in 201 1 — 5 years behind the originally scheduled date of 2006 — is not finalized 
because of the continuing contract item renegotiations and decisions regarding vessel 
asset tracking functionality. Finally, these additional factors may increase costs: (1) the 
variability of costs for tower preparation and construction and (2) site reorganization. 2 
USCG realizes that it needs to be more effective in managing costs and schedule and 
has plans to complete an integrated baseline review for the contract items as these are 
renegotiated; however, a date for completing this review has not been established. In 
addition, USCG plans to begin monthly reviews of the project schedule and to 
incrementally deploy to future Rescue 21 sites. 



2 USCG is in the process of reorganizing its field units into new integrated and standardized organizations, referred to as sectors. 
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Accountability * Integrity * Reliability 



The delays in deploying Rescue 21 could have several effects: 



■ key functionality, such as improved direction finding and improved coverage, will not be 
available for a significant period of time; 



■ legacy equipment may fail, requiring costly repairs; and 



■ legacy equipment may require upgrades to address coverage gaps and other 
operational issues. 

We are recommending that the Secretary of Homeland Security direct the Commandant 
of the United States Coast Guard to ensure that executive-level management oversees 
the project's progress toward cost and schedule milestones and manages risks. In 
addition, we recommend that executive-level management establish a milestone to 
complete its integrated baseline review and revise its cost and schedule estimates. 

In providing oral comments on a draft of this briefing, the Coast Guard Assistant 
Commandant for Acquisition stated that the briefing fairly represented Rescue 21 risks 
and ongoing activities to address program management and other issues. He also stated 
that this briefing would be used as input to take corrective actions to improve program 
management. 
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Background 
Rescue 21 Description 

The NDRS is USCG's primary VHF communications system, important to the USCG's 
search and rescue program as well as its homeland security mission. However, the USCG 
has reported that NDRS — much of which was installed in the 1970s — has several critical 
deficiencies: 

■ numerous gaps in communications coverage — in some areas the Coast Guard cannot 
hear calls from mariners in distress or communicate with other USCG vessels; 

■ limited direction-finding capabilities and no digital selective calling capabilities; 3 

■ no effective way to track USCG assets that conduct search and rescue; 

■ limited interoperability with other federal and state communications systems; and 

■ no means for protected communication. 

Rescue 21 's primary functions are to provide USCG with communication for maritime 
search-and-rescue functions — such as monitoring distress calls from boaters and 
coordinating the response or rescue. It is intended to replace NDRS, and will be a short- 
range communications system reaching about 20 nautical miles. The system will include 
VHF-FM radios, communications towers, and hardware and software at USCG sites as well 
as on Coast Guard vessels. This system also, in addition to search and rescue, provides 
support to homeland security missions. USCG's acquisition directorate, under the Assistant 
Commandant for Acquisition, is responsible for managing the Rescue 21 acquisition. 



k G A O 
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3 A digital selective calling radio has the ability to send a mayday signal that identifies the vessel and, when connected to a Global Positioning System, sends the vessel's 
location. This occurs on channel 70. 



Page 16 



GAO-06-623 Coast Guard's Rescue System 



Appendix I: Briefing Slides 



Background 
Rescue 21 Description 

In September 2002, USCG awarded an indefinite delivery, indefinite quantity 4 contract to 
General Dynamics for developing and implementing Rescue 21. The contract and original 
schedule called for the following: 

■ two initial Coast Guard regions to operate the new system with full functionality — 
referred to as initial operating capability — scheduled for September 2003 and 

■ continued deployment to additional locations through 2006. 

According to the Coast Guard contract with General Dynamics, the Rescue 21 system is to 

■ allow continuous, uninterrupted communications on Channel 16; 5 

■ limit the number of communication coverage gaps to less than 10 percent in the 
United States (originally this was to less than 2 percent); 

■ provide direction finding and digital selective calling to locate boaters; 

■ allow USCG to track its mobile assets, such as vessels; 

■ allow communication with other federal and state systems; 

■ allow protected communication of sensitive information; and 

■ archive voice and data communications and make them instantly accessible. 

A table showing the capabilities of NDRS and Rescue 21 follows. 



k G A O 
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4 An indefinite-delivery, indefinite-quantity contract provides for an indefinite quantity, within stated limits, of supplies or services during a fixed period of time. The government schedules 
deliveries or performance by placing orders with the contractor. 48 C.F.R. §16.504 "| 

5 According to USCG officials, Channel 1 6, which is analogous to 91 1 , is the channel designated for the use of distressed boaters to gain assistance in an emergency. 
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Background 
Rescue 21 Planned Capabilities 



Comparison of Rescue 21 Capabilities with Legacy System 





Capabilities 


Existing NDRS 


Rescue 21 


Monitor 
distress calls 


Continuous, uninterrupted 
Channel 16 Guard 


no 


yes 


Channel 70 digital selective calling and 
direction finding 


no 


yes 


Communications coverage 


Unknown; numerous 
gaps 


90-98% coverage to 20 
nautical miles from the 
shore 


Alert response 
assets 


Automatic vessel asset tracking 


no 


to be determined 


Data communications (between vessels and stations) 


no 


to be determined 


Coordinate 

response 

activities 


Public safety interoperability 


no 


yes 


Full coverage protected communications 


no 


yes 


Geographic display 


no 


yes 


Number of simultaneous communications channels 


1 


6 



Source: U.S. Coast Guard. 
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Background 
Rescue 21 Components 

Key components of Rescue 21 — some of which appear in the following diagram — are 

■ communications centers (46) that contain operator workstations and electronic 
equipment; 

■ antenna towers (322); 

■ other equipment and/or infrastructure that could be used on vessels or remote 
facilities, including communication lines and radio equipment; and 

■ disaster recovery assets (not pictured). A disaster recovery asset is a smaller antenna 
tower and equipment similar to a remote facility. It is portable so that it can be moved 
to locations as needed. 
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Background 
Rescue 21 Components 



Group communications 
center processes distress 
signal and directs rescue 



High level site towers 
determine direction 
of signal 




Source: GAO analysis of U.S. Coast Guard data. 
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In September 2003, we reported that USCG's original schedule for reaching initial 
operating capability by September 2003 had been postponed and that it had yet to finalize 
a new schedule. 5 This postponement was due, in part, to the development of the system 
taking longer than planned. 

We also determined that USCG postponed key tests and the initial operating capability 
because of the delays in software development, and we highlighted risks associated with 
its plan to compress and overlap key tests as a result of the delays. In addition, schedules 
for key tests and deliverables, including test plans, were still not complete. 

We recommended that the Secretary of the Department of Homeland Security direct the 
Commandant of the United States Coast Guard to 

■ establish a new schedule for critical testing phases and initial operating capability and 

■ ensure that milestones were established for completing test plans for formal 
qualifications testing, system integration testing, and operational testing and 
evaluation and that test plans address all requirements for the system. 

Following our report, key test schedules were revised and milestones and test plans were 
developed. 



5 GAO, Coast Guard: New Communications System to Support Search and Rescue Faces Challenges, GAO-03-1 1 1 1 (Washington, D.C.: Sept. 30, 2003). 
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However, further delays occurred during testing. 6 For example, 

■ Formal qualification testing, originally expected to begin in February 2003 and end in 
March 2003, actually began in January 2004 and ended in June 2004 — 15 months 
behind schedule. At that time, only 4 of the 235 allocated requirements were validated 
and performance issues in several areas, including fault management, archive/restore, 
and channel performance, were found that required additional work. Formal qualification 
testing was repeated in June 2004, at which time the remaining 231 allocated 
requirements were validated. 

■ System integration testing, originally expected to begin in April 2003 and end in May 
2003, actually began in August 2004 and ended in December 2004 — 19 months behind 
schedule. Additional software defects were found during this time and USCG decided to 
have the contractor concurrently fix the system integration testing problems while going 
into the operational test and evaluation phase. 




GAP 

Accountability * Integrity * Reliability 



6 Rescue 21 tests include: formal qualification testing, which demonstrates that the system meets contract performance specifications; system integration testing, which demonstrates 

that the system is fully functional at USCG stations; and operational testing and evaluation, which is conducted by the USCG and ensures that the system satisfies contract performance 1 5 

requirements and is ready to be deployed at other USCG locations. 
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Rescue 21 Schedule Delays 



■ Operational test and evaluation, originally intended to begin in July 2003 and to end in 
August 2003, actually began in January 2005 and ended March 2005 — 19 months 
behind schedule. Further software defects were found, and USCG also requested 
additional changes to the human systems interface based on operator comments. The 
changes to the human systems interface were prioritized into three builds, the first of 
which was required by USCG before it would approve initial operating capability. The 
remaining two builds are being negotiated with the contractor and represent additional 
development. Finally, it was determined that the vessel portion of Rescue 21 did not 
meet performance requirements. 



Because of the delays in development and testing, the date for initial operating capability 
was delayed from September 2003 to December 2005. Full operating capability, which 
was originally scheduled for fiscal year 2006, was delayed to fiscal year 201 1 . 
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Rescue 21 Schedule Delays 

Development and Implementation Schedule 



Development 



FQT 


D 








SIT 


D 




1 




OT&E 


D 






1 



IOC 

I 



Implementation 



i T 

IOC FOC 
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2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 



Original schedule 

Actual or projected schedule 



Source: GAO analysis of U.S. Coast Guard data. 
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Background 
Rescue 21 Cost Growth 

Rescue 21 's total acquisition cost estimate has increased from $250 million to $710.5 million 
over a 6-year period, as shown in the following chart. 
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April '99 November '99 March '02 June '05 

Acquisition Cost Totals 



Source: U.S. Coast Guard Acquisition Project baseline data. 
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Background 
Rescue 21 Appropriations 

Rescue 21 has been appropriated a total of $382.9 million for development since award of 
the contract in fiscal year 2002. 
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Source: U.S. Coast Guard data. 
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Rescue 21 was deployed on December 20, 2005, with an initial operating capability 7 and is 
being used at two regions (Atlantic City, New Jersey, and Chincoteague, Virginia). The 
deployment included all the station functionality and equipment that 

allowed continuous, uninterrupted communications on Channel 16; 
limited the number of communication coverage gaps; 
provided direction finding and digital selective calling to locate boaters; 
allowed radio communication with other federal and state officials; 
allowed protected communication of sensitive information; and 
archived voice and data communications and made them accessible in real time. 



However, functionality for vessel asset tracking and data transfer plus an expanded vessel 
communications suite was not delivered as planned. This was deferred because the 
contractor could not meet the original contract requirements for simultaneous 
communications. In January 2006, a stop work order was issued on the vessel tracking and 
data transfer functionality for 90 days. USCG and the Department of Homeland Security 
(DHS) are currently determining whether Rescue 21 should complete development of the 
vessel functionality or consider other sources for that functionality. 



7 USCG also uses the term "ground sub-system" for the functionality and equipment deployed to the stations in the initial operating capability regions. The functionality and equipment 
that was to be deployed to the vessels associated with these stations, which was not delivered as scheduled, is referred to as the "vessel sub-system". 
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The current schedule calls for the deployment of Rescue 21 to four low-rate initial 
production sites during 2006. 8 In addition, according to USCG officials, they are currently 
revalidating implementation plans for the first 15 of the remaining 40 sites. These first 15 of 
the remaining 40 sites are approved for deployment and are currently scheduled to be 
completed by fiscal year 2009. Completion of the remaining 25 sites with full operating 
capability is currently scheduled for fiscal year 201 1 . 

However, some development activity remains. First, there are two remaining builds of 
enhancements to the human systems interface, with the second build scheduled for 
completion in October 2006 and the third in March 2007. Second, a decision is pending on 
how to implement vessel asset tracking and data transfer functionality. 




Page 28 



GAO-06-623 Coast Guard's Rescue System 



Appendix I: Briefing Slides 



1 G AO Objective 1 

Accountability * Integrity * Reliability 

Reasons for Cost Overruns and Schedule Delays 

Key factors that contributed to Rescue 21 cost overruns and schedule delays were 
shortcomings in requirements management, project monitoring, risk management, 
contractor cost and schedule estimation and delivery, and executive-level oversight. USCG 
agreed that improvements need to be made in these areas and is taking steps to address 
these issues. 



To effectively manage major IT programs, organizations should use sound acquisition and 
management practices to minimize risks and thereby maximize a project's ability to be 
delivered on time, within budget, and with promised functionality. Such practices have been 
identified by leading organizations, such as the Software Engineering Institute, the Chief 
Information Officer's Council, and in our prior work analyzing best practices in industry and 
government. 
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Requirements management 

Requirements management includes processes for eliciting, documenting, verifying and validating, and 
managing requirements through the system life cycle. 



The Coast Guard did not have a rigorous requirements management process. Although the 
USCG developed high-level requirements for Rescue 21, it relied solely on the contractor 
to manage these requirements. According to USCG acquisition officials, they took this 
approach because of the performance-based contract vehicle. However, our experience in 
reviewing major systems acquisitions has shown that it is important for government 
organizations to exercise strong leadership in managing requirements, regardless of the 
contracting vehicle. 

In addition to not effectively managing requirements, testing revealed numerous problems 
linked back to incomplete and poorly defined user requirements. For example: 

■ A USCG usability and operability assessment of Rescue 21 stated that most of the 
operational advancements envisioned for the system were not achieved and 
concluded that these problems could have been avoided by including user 
requirements in the contract. 
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Reasons for Cost Overruns and Schedule Delays 

Requirements management (continued) 

■ A key requirement stated, "Provide a consolidated regional geographic display." The 
contractor provided a capability based on that statement, but during testing the USCG 
operators felt the maps did not display sufficient detail. Such discrepancies led to an 
additional statement of work defining required enhancements to the system interface 
(e.g., screen displays). 

After these initial problems, USCG recognized the need for an orderly process to review 
changes that could have a significant impact to cost, schedule, and performance. 
Specifically, in October 2005, USCG established a Configuration Control Board to control 
changes to Rescue 21 requirements. 
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Project monitoring 

A key project monitoring activity uses earned value management (EVM) techniques to track the progress of 
work completed against project cost and schedule estimates. 



USCG did not effectively use EVM data in order to measure performance and take 
corrective action. The Coast Guard has received EVM cost performance reports that 
identified problems on a regular basis from Booz Allen Hamilton. 9 However, USCG did not 
take action on these reports. For example, the March 2003 report indicated that, given the 
number of slipped tasks and small remaining time, it was unlikely that the contractor would 
be able to recover the schedule deadlines; the September and October 2003 reports 
described the contractor's performance as rapidly worsening. Nevertheless, USCG did not 
take action in response to these reports. 



9 USCG contracted Booz Allen Hamilton to assist with program management for Rescue 21 . 
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Risk management 

An effective risk management process identifies potential problems before they occur, so that risk-handling 
activities may be planned and invoked as needed across the life of the product and project in order to 
mitigate adverse impacts on achieving objectives. Key activities include identifying and analyzing risks, 
assigning resources, developing risk mitigation plans and milestones for key mitigation deliverables, 
briefing senior-level managers on high-priority risks, and tracking risks to closure. 



USCG's management process includes identifying, analyzing, and prioritizing risks; 
establishing mitigation strategies; and assigning responsibility for addressing risks. 
However, USCG did not always mitigate and communicate risks. For example: 

■ In 2003, USCG identified software development as a high risk to Rescue 21 's 
schedule. However, despite early acknowledgment of this risk, it was not mitigated 
and ultimately became one of the primary drivers for the increase in costs. 

■ USCG did not communicate risks to partners when applicable. During our review, the 
contractor stated that it was unaware of the items in the USCG risk list that related to 
the contractor's performance. 
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Contractor cost and schedule estimation and delivery 

The contractor underestimated the time for development and testing of Rescue 21 and 
established an integrated project schedule that it could not meet. 

■ Development took longer than planned, which led to delays in testing. Subsequently, 
testing phases were not completed on schedule. 

■ Additionally, the initial estimate for completing a regional site deployment was 12 
months. The new estimate of 24 months is considered to be more realistic for 
executing the same tasks. 

The contractor did not always provide personnel in a timely manner or with the knowledge 
and skills required to complete the project. 

■ According to USCG information, it took longer than expected to obtain developers and 
subcontractors for Rescue 21 . 

■ According to USCG documentation, support personnel involved in initial site surveys 
were minimally qualified and had limited civil engineering and environmental analysis 
skills. 
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Executive oversight and governance 

Office of Management and Budget and GAO guidance call for agencies to establish IT investment 
management boards comprised of key executives to regularly track the progress of major systems 
acquisitions. These boards should have documented policies and procedures for management oversight of 
IT projects and systems and should be able to adequately oversee the project's progress toward cost and 
schedule milestones and their risks. The board should also employ early warning systems that enable it to 
take corrective actions at the first sign of cost, schedule, and performance slippages. 



USCG told us that it had an executive oversight process that included semi-annual and key 
decision point reviews. However, there is no evidence that these reviews of Rescue 21 
occurred before 2005. 



Rescue 21 oversight meetings commenced in 2005, but executive management did not 
take action in response to the risks and problems presented. For example, briefings given 
to the USCG Acquisition Review Council and the Department of Homeland Security (DHS) 
Investment Review Board for key decision point 3 approval identified numerous high-level 
and moderate risks and DHS officials requested additional actions, but executive-level 
monitoring to ensure the adequacy and completion of risk mitigation activities did not occur. 
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Reasons for Cost Overruns and Schedule Delays 

Recent Efforts by USCG to Improve Rescue 21 Management 

USCG has recently begun to address several of the key factors that led to cost overruns and schedule 
slippages. 

According to the Assistant Commandant for Acquisition, the USCG plans to improve the capabilities of 
the Acquisition Office by obtaining more technical assistance to help manage requirements, monitor 
project performance, and mitigate risks, among other things. For example, USCG intends to begin 
monthly reviews of the project schedule in June 2006 to track actual against planned milestones and to 
use earned value management data to improve its oversight of cost and schedule performance. 

Regarding contractor performance, USCG stated that it was difficult to manage the project effectively 
from headquarters and that the distance between USCG and the contractor delayed decisions. As a 
result, in November 2005, USCG established a Project Resident Office adjacent to the contractor's 
facility. This office's responsibilities include overseeing nationwide regional site implementation; 
monitoring contractor performance; managing risks; establishing closer communication and coordination 
with the contractor to minimize contract changes and control cost growth; and monitoring shipping and 
delivery of Rescue 21 equipment. 

USCG has also increased the visibility of Rescue 21 at the executive level by establishing quarterly 
meetings between the USCG Assistant Commandant for Acquisition and the vice president of the 
contractor. These meetings may facilitate greater executive-level involvement in Rescue 21. 
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Coast Guard's current acquisition cost estimate of $710.5 million is not viable. Our analysis 
of contractor performance trends indicates that there will likely be additional cost overruns 
that will bring the total acquisition cost to $872 million unless critical changes are made. 
This includes a significant number of contract items that have not been completed as 
planned and must now be renegotiated. Also, USCG's schedule is uncertain and will likely 
change due to the continuing contract item renegotiations and decisions regarding vessel 
functionality. Negotiations for contract items for the first 15 full-rate production regions 
continues and is expected to be completed by June 2006. Subsequently, contract items to 
deploy Rescue 21 to the remaining 25 full-rate production regions must also be 
renegotiated. 

Finally, additional factors may increase costs, such as the variability of costs for tower 
preparation and construction and site reorganization. 10 
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As of June 2005, the current acquisition cost estimate to reach full operational capability in 
2011 for Rescue 21 is$710.5 11 million. 

Based on our analysis of contractor performance trends (using contractor EVM data 12 ), we 
forecast a cost overrun of approximately $161 million. The total acquisition costs could be 
as high as $872 million. 

The current cost estimate does not take into account contract items that are being 
renegotiated. There are 91 contract items for site deployment that have expired and have 
to be renegotiated. According to USCG, based on their experience with the cost overruns 
at the IOC sites, costs for each of the contract item listed below will increase: 

■ regional implementation plans 

■ regional infrastructure preparation 

■ regional system installation 

Coast Guard officials agreed with our analysis of projected cost overruns. 



11 This estimate reflects only the acquisition component and does not include operations & maintenance. The total life cycle cost estimate is approximately $1 .6 billion. 

12 We used contractor- provided data that reflected work from October 2003 to December 2005. 
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Objective 2 

Revised Cost and Schedule Estimates 



USCG's schedule is also uncertain and will likely change. 

■ The current project schedule is uncertain because of the 91 contract items that must 
be renegotiated. USCG estimates that this renegotiation will be completed for its first 
15 full-rate production regions in June 2006. Contract items for the remaining full-rate 
production regions — intended to be addressed in two additional groups — are to be 
negotiated at a later date. 

■ There is also uncertainty regarding Rescue 21 vessel functionality. Initial plans for 
Rescue 21 included functionality to allow USCG to track its vessels and to allow data 
transfer between vessels and group communications centers plus an expanded 
vessel communications suite. This functionality is now being deferred while USCG 
and DHS decide whether to complete development of its contractor-developed 
solution or use a different solution such as the Automatic Identification System 13 for 
asset tracking and data transfer on vessels. This decision is currently expected by the 
end of April 2006. 



,3 The Automatic Identification System is a Coast Guard project to develop a new system that will be used to monitor and track vessels traveling to and through US waters. 
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Additional issues may also contribute to further cost overruns and schedule delays. 

Variability in infrastructure preparation costs. According to USCG officials, there is 
significant variation in total costs for antenna towers. Additional costs can be incurred in 
building a tower and significant schedule delays can occur. Further, environmental 
concerns may further delay tower construction. For these reasons, there is significant 
potential for cost overruns and schedule slippages in tower construction. 

Site reorganization. New site command centers are to be established throughout USCG, 
based on a new standard organization model. There is no guarantee that current sites will 
have the same functions that they have now. For example, the Eastern Shore group will 
become a site with the communication center at Hampton Roads. The equipment at the 
Eastern Shore GCC will be moved to Hampton Roads and sites will be reconfigured to the 
new arrangement. Therefore, rework may be needed when the site centers are finalized. 
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Cost and Schedule Management 

USCG has plans to improve its cost and schedule management in the future. For example: 

■ USCG intends to conduct an integrated baseline review of the contractor's cost and 
schedule estimates after it completes renegotiation of expired contract items for the 
deployment to the first 15 full-rate production regions. This baseline review is currently 
expected to begin in the June 2006 time frame and is supposed to validate the 
accuracy of the Rescue 21 schedule. However, there is no completion date for the 
baseline review. USCG also intends to complete another baseline review of the 
schedule once the contracts for the remaining 25 full-rate production sites are 
renegotiated. 

■ USCG intends to conduct full-rate production site deployment incrementally as it 
expects this gradual deployment to improve its ability to manage cost and schedule. 
The contracts for the first group of 15 regions are currently being negotiated; contracts 
for the second set of 1 1 regions and third set of 14 regions will be addressed 
afterwards. 
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Delay in Implementation 

The effects of the delays in deploying Rescue 21 include (1) key functionality, such as 
improved direction finding and improved coverage, will not be available; (2) potential failure 
of legacy equipment requiring costly repairs; and (3) potential for further upgrades to the 
legacy system and resulting costs, despite a moratorium on upgrades. 

Key functionality to production sites will be delayed. Originally, Rescue 21 was to be 
deployed nationwide by fiscal year 2006. The current schedule shows full deployment in 
fiscal year 201 1 , five years behind schedule. This will result in a significant implementation 
delay for many field units. Personnel at the two initial operating capability stations 
acknowledged that Rescue 21 provides significantly improved functionality compared to the 
legacy system. For example, according to these personnel, Rescue 21 's direction finding 
capability — which allows USCG to locate boaters in distress-is significantly more accurate 
as well as more reliable than the legacy system. As a result, until Rescue 21 is deployed, 
field units with the legacy system will continue to be at risk of performing larger and 
potentially more costly searches due to limited direction finding capabilities. 
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Objective 3 
Delay in Implementation 



Until Rescue 21 is deployed, field units will likely continue to experience communication 
coverage gaps, which limits their ability to hear boaters in distress. However, the impact of 
coverage gaps varies depending on the availability of local partners to conduct search and 
rescue, or vessels to relay distress calls. Personnel at one field unit explained that they rely 
on a marine patrol unit to conduct searches in a coverage gap area, whereas at another 
site, the USCG field unit is the only resource available to conduct search and rescue 
operations. Nevertheless, until Rescue 21 is fully deployed, missed mayday calls — and the 
potential for lives lost — creates uncertainty about the legacy system's reliability and 
reduces operators' confidence in the system. 

Potential for failure of legacy equipment requiring costly repairs. The legacy system is 
about 30 years old and is difficult to maintain. Until Rescue 21 is deployed, legacy 
equipment will continue to be at high risk for failure; these failures could result in costly 
repairs. Some USCG maintenance officials questioned whether legacy equipment will 
continue to function until Rescue 21 is fully deployed, especially in field units scheduled for 
later implementation. Field units could also experience higher costs resulting from the need 
to replace equipment and reengineer parts that are no longer manufactured. In addition, 
maintenance expertise needed to repair the antiquated equipment is limited and, as a 
result, repairs are not always timely. 
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Further upgrades to the legacy system to address operational issues could occur, 
despite a moratorium on upgrades. The USCG instituted a moratorium on upgrades to 
the legacy system because Rescue 21 was to replace the equipment. However, with units 
facing delays of up to 5 years, USCG may decide that the risk to its operations is 
sufficiently high that it should upgrade or install new legacy equipment rather than wait to 
receive the new system. In certain cases, this has already occurred, despite the 
moratorium. For example, USCG officials reported that they upgraded a console at one unit 
to mitigate operational challenges, and installed a new antenna at a second unit to address 
coverage gaps. Additional upgrades such as these would result in higher costs, especially 
at field units scheduled for later implementation. 
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Conclusions 

Accountability * Integrity * Reliability 

Although USCG plans to improve its capabilities in a number of areas to better manage the 
remaining development and deployment of Rescue 21 by establishing executive-level 
meetings with the prime contractor, it is imperative that key executives aggressively 
monitor costs and schedule performance and mitigate risks so the remaining development 
and deployment of Rescue 21 is effectively managed and delivered. 



USCG's current acquisition cost estimate is not viable and its schedule is uncertain and 
likely to change. USCG has plans to complete an integrated baseline review, but it has not 
established a target completion date. Without the results of this review, the cost and 
schedule estimate cannot be completed. 

If deployment of Rescue 21 continues to be delayed, Coast Guard sites and services will 
be impacted in several ways. First, key functionality, such as improved direction finding and 
improved coverage of coastal areas, will not be available as planned. Coast Guard 
personnel at these sites will continue to use outdated legacy communications systems for 
search and rescue operations and coverage of coastal regions will remain limited. In 
addition, these delays could result in costly upgrades to the legacy system to address 
communications coverage gaps as well as other operational concerns. 
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Recommendations 



To more effectively manage the remaining development and deployment of Rescue 21 , we 
recommend that the Commandant of the United States Coast Guard ensure that executive- 
level management implements the following three recommendations: 



■ oversee the project's progress toward cost and schedule milestones and manage 
risks, 

■ establish a milestone to complete Rescue 21 's integrated baseline review, to include 
all renegotiated contract items, and 

■ use the results of this review to complete a revised cost and schedule estimate. 
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Agency Comments 



In providing oral comments on a draft of this briefing, the Coast Guard Assistant 
Commandant for Acquisition stated that the briefing fairly represented Rescue 21 risks and 
ongoing activities to address program management and other issues. He also stated that 
this briefing would be used as input for them to take corrective actions to improve program 
management. 

Regarding the status of the vessel asset tracking and data transfer functionality that was 
deferred, the Assistant Commandant stated that he expects this functionality to be 
addressed with the Automatic Identification System, rather than by Rescue 21 . This 
decision is pending Department of Homeland Security concurrence. 

Coast Guard did not concur with our statements regarding executive oversight. They stated 
that, prior to 2005, there were briefings to executive management, such as the key decision 
point and semi-annual meetings. However, as previously stated, there is no evidence that 
these executive-level meetings occurred. 

The Coast Guard also provided specific technical comments, which we have incorporated 
into the briefing as appropriate. 
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U.S. Department of Homeland Security 

Washington, DC 20528 




Homeland 
Security 



May 12, 2006 



Mr. David A. Powner 
Director 

Information Technology Management Issues 
U.S. Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Mr. Powner: 

Thank you for the opportunity to review and comment on the Government 
Accountability Office's (GAO's) draft report entitled UNITED STATES COAST 
GUARD: Improvements Needed in Management and Oversight of Rescue System 
Acquisition (GAO-06-632). This draft report includes briefing slides presented on 
March 27, 2006 to the House Committee on Appropriations, Subcommittee on 
Homeland Security and the Senate Committee on Commerce, Science and 
Transportation, Subcommittee on Fisheries and Coast Guard. 

While we agree with GAO's three recommendations contained therein, and are in the 
process of implementing each recommendation, we do have substantial concerns with 
several parts of the report and the briefing. 

We agree that the Coast Guard is responsible for managing schedule delays and cost 
overruns in major acquisition projects. The following program management 
improvements to control cost and schedule have been initiated by the Department of 
Homeland Security (DHS) and the Coast Guard: 

• Established an on-site Coast Guard Project Resident Office for increased 
contractor oversight. (Aug 2005). 

• Ordered full rate production regions incrementally to improve control (Sep 2005). 

• Implemented Alpha Contracting for improved communications between the 
government and contractor to streamline the process (Sep 2005). 

• Implemented fixed price maintenance contracts after initial year (Sep 2005). 



www.dhs.gov 
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• Increased oversight with quarterly Flag Officer/Vice President level program 
reviews and periodic DHS Joint Requirements Council (JRC) and Investment 
Review Board (IRB) reviews (Oct 2005). 

• Issued a Terminate for Convenience of the Government letter to the contractor on 
all vessel initiatives as a cost mitigating strategy (April 2006) 

• Will conduct Integrated Baseline Reviews (IBR) for Full Rate Production to 
better identify and control cost beginning Aug 2006. 

• Will conduct monthly reviews of project schedule with the contractor beginning 
June 2006. 

As reflected above, many of the cost and schedule concerns raised by the report have 
since been addressed. Accordingly, the project's APB (acquisition program baseline) for 
cost, schedule, and performance have been revised. For instance, the total acquisition 
cost parameter is S730.2M (vice $872M, as estimated by GAO) which has been achieved 
by adjusting several performance parameters. Additionally, on April 21, 2006, the Coast 
Guard issued a notice of termination for convenience, canceling General Dynamics' 
option to provide all vessel sub-system (VSS) components. The Coast Guard anticipates 
that this work will be handled organically or by another contractor or a combination of 
both. 

The shortcomings cited in the GAO report, however, do not adequately emphasize the 
root cause of the RESCUE 21 system deployment delays and overruns. As confirmed by 
GAO's findings on page 27 of the briefing, the contractor: 

• underestimated the time for development and testing of RESCUE 2 1 and could 
not meet its own established integrated project schedule, and 

• failed to consistently provide personnel in a timely manner or with the knowledge 
and skills required to complete the project. 

General Dynamics' inability to meet the technical performance within its proposed 
schedule was due to challenges in software development. The Coast Guard was fully 
aware of the problem the contractor was experiencing and made a conscious decision 
during the software development phase to offset cost and schedule to ensure technical 
performance was not compromised for this critical life-saving system. 

The GAO draft report highlights five key factors that contributed to RESCUE 21 cost 
overruns and schedule delays. As noted above, the Coast Guard agrees with GAO's 
fourth factor regarding "Contractor Cost and Schedule Estimation and Delivery." 
However, the remaining key factors are not accurate in important aspects and the Coast 
Guard provides the following facts and rationale (the following citations reflect the 
revisions GAO made to the draft report as a result of subsequent discussions with the 
Coast Guard): 
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Requirements Management: GAO "Coast Guard did not effectively manage 
requirements and testing revealed incomplete and poorly defined requirements." 

This draft report does not recognize well-established Coast Guard processes and is not 
consistent with the GAO's previous acknowledgement of these processes (Report GAO- 
03-1 111). The Coast Guard's requirements management process is outlined in the Major 
Systems Acquisition Manual (MSAM COMDINST M4150.2) Chapter 3 "Requirements 
Management Process," which includes development of the: 

• Mission Capability Gap Analysis documented in the Mission Analysis Report 
(MAR) dated 1 9 May 1 994. 

• Broad Mission and Capability Need documented in the Mission Needs Statement 
(MNS) dated 18 Apr 1995, updated 14 May 1998, and most recently revalidated 
12 Apr 2005. 

• Mission Oriented Capabilities documented in the Preliminary Operational 
Requirements Documents (PORD) dated 19 Nov 1997, and the Operational 
Requirements Document (ORD)1dated 19 Jul 1999, updated 20 Aug 2001, and 
most recently revalidated 12 Apr 2005. 

• System Specific Capabilities documented in the Performance Specification (P- 
Spec) dated 8 Feb 2002. 

The Coast Guard maintains that such requirements as, "receive a 1 watt transmission 
from 2 meters above the water 20NM offshore" and "Direction finding (DF) to within +/- 
2 degrees", demonstrate the specificity of the RESCUE 21 system requirements, as 
documented in the Performance Specification. 

Moreover, field testing demonstrated that the requirements were clearly defined and that 
such testing is the appropriate mechanism to verify whether General Dynamics' design 
met the contractual requirements of the Performance Specifications. 

• General Dynamic's software design (and to a certain degree the contractor's 
hardware configuration design) included over 200 defects that required correction 
and re-testing for conformance to requirements prior to acceptance by the Coast 
Guard. These defects were documented in the Formal Qualification Testing 



With respect to the ORD, the GAO draft report (Page 2) states, "Also, originally, Rescue 2 1 was intended 
to limit communications gaps. However, recent information indicates that while the Coast Guard has a 
goal of achieving 2 percent gaps to the extent possible, constraints may limit their ability to meet this goal. 
Coast Guard further stated that limiting gaps to 2 percent was only a goal, its threshold gap level was 10 
percent or less." The Coast Guard has not made any reduction in coverage requirement since the ORD 
(Operational Requirements Document) was approved in 2001. The ORD specifies a communications 
coverage goal of 98% and threshold of 90%. The goal of the Coast Guard is to achieve 98% 
communications coverage to the maximum extent possible. In discrete cases, where environmental, 
historical preservation, geography or other external factors may constrain communication coverage to 
something less than 98%, the established threshold of 90% as documented in the ORD will be met. In no 
case will the project deploy less than 90% communications coverage unless a waiver is approved (only 
considered in rare situations where the cost of covering a particular area is grossly disproportionate to the 
potential benefits). 
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(FQT) and System Integration Testing (SIT) plans; both of which were provided 
to GAO. 

• While General Dynamics ultimately met the system requirements as a result of 
their efforts to correct defects and retest performance, the Coast Guard determined 
that the functionality of the system could be improved by incorporating human 
systems interface (HSI) enhancements proposed by the users. Given the life- 
saving nature of the RESCUE 21 system, the Coast Guard believes that (he 
decision to offset schedule for greater HSI performance was prudent, and in the 
end ensures the Coast Guard can adequately serve the American public. 

Project Monitoring: GAO "Coast Guard did not effectively use earned value 
management data to measure performance and take corrective action on negative 
trends." 

Since award of the Phase II contract to General Dynamics in September 2002, the Coast 
Guard established an Earned Value Management (EVM) work group comprised of Coast 
Guard officials with support from Subject Matter Experts from Booz Allen Hamilton. 
Together, this group was responsible for reviewing the monthly Cost Performance 
Reports (CPR) submitted by General Dynamics, conducting analyses to determine 
performance trends and developing monthly Performance Measurement Reports. 

In accordance with the performance-based, performance-specification nature of the Phase 
II contract, the RESCUE 21 Project Manager (PM) employed the recommendations of 
the Coast Guard EVM work group to highlight to General Dynamics' upper management 
their poor performance (e.g., inability to maintain cost and schedule performance within 
the variance thresholds). The PM further requested that General Dynamics institute 
corrective actions to reverse the negative performance trends. General Dynamics clearly 
acknowledged the challenges they were facing in designing the system as documented in 
their Nov 2002 CPR: 

From General Dynamics Source Document ID: PIA 16276/A008-002A (Nov 2002): 

• "The schedule variance was primarily attributable to the Software Development 
account" 

• "The program's critical path continues to be driven by the software development 
effort..." 

The following excerpts are from formal correspondence the Coast Guard submitted to 
General Dynamics and this information was provided to GAO as evidence of concrete 
actions taken by the RESCUE 21 PM in light of the EVM data: 

From Coast Guard Source Document ID: PIA 16276/A008-01 7 (Feb 2004): 

• "The CG is extremely concerned about the current and cumulative cost/schedule 
performance trends and status related to the cost plus CLINs. There are several 
key issues that must be addressed in order to provide insight into the contractor's 
commitment to control cost overruns and recover from schedule delays." 
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• "The variance reporting must be expanded to provide more insight into ongoing 
and future risk mitigation and recovery plans related to cost control and schedule 
recovery." 

From Coast Guard Source Document ID PIA 16276/A008-022 (Jul 2004): 

• 'The USCG is very concerned that the information reflected on the CPR is not 
being used by GDDS as a management tool to make informed decisions and 
manage risk." 

In response to Coast Guard concerns, in June 2004 General Dynamics undertook a 
significant re-organization of its management structure as a corrective action in an 
attempt to better manage cost and schedule performance. 

Risk Management: GAO "Coast Guard did not effectively mitigate and 
communicate risks." 

In addition to the process for managing risks as documented in RESCUE 21 's Risk 
Management Plan, risks are discussed and reported during the Project Management 
Reviews (PMR), Semi-Annual briefs, Performance Measurement Reports, Monthly 
Acquisition Reports, Coast Guard Acquisition Review Council (CGARC) and IRB 
meetings, and various weekly/monthly workgroup meetings. During each PMR, risks 
affecting cost, schedule, logistics, and technical performance are communicated to all 
General Dynamics and Coast Guard teams and follow on discussions are held to validate 
the mitigation actions, ownership, and planned dates for closure. 

For example, the risk associated with software development was communicated, and 
actions were taken to mitigate 'TECHNICAL PERFORMANCE" risk. This risk was 
assigned top priority, given the life-saving aspect of RESCUE 21. Mitigation actions 
included (i) establishment of an HSI workgroup comprised of Coast Guard and General 
Dynamics personnel to discuss and resolve issues, (ii) assignment of a member of the 
Coast Guard team to the General Dynamics' facility for 6 months to facilitate issue 
resolution, (iii) establishment of a contract modification to include enhancements to the 
system proposed by the Coast Guard operational community, (iv) development of a plan 
of action and milestones to implement the HSI enhancements. 

Executive-Level Oversight: GAO "Coast Guard stated that it had an executive 
oversight process that included semi-annual and key decision point reviews. 
However, there is no evidence that these reviews of Rescue 21 occurred before 
2005." 

Early in the audit, the Coast Guard provided oral evidence as to the existence of senior 
leader executive oversight, upon which GAO requested documentary evidence. The 
Coast Guard later provided GAO staff a CD containing the semi-annual and key decision 
reviews dating from 2002 thru 2005. These reviews included: 



Page 52 



GAO-06-623 Coast Guard's Rescue System 



Appendix II: Comments from the Department 
of Homeland Security 



6 



Jan 2002 


Key Decision Point 2 (milestone decision) 


Oct 2002 


Semi- Annual Presentation to Coast Guard Acquisition Review Council (CGARC) 


Aug 2003 


Semi- Annual Presentation to CGARC 


Feb 2004 


Semi- Annual Presentation to CGARC 


July 2004 


Semi- Annual Presentation to CGARC 


Apr 2005 


Key Decision Point 3 CGARC 


Jun 2005 


Key Decision Point 3 DHS Investment Review Board 



Additionally, throughout this period the RESCUE 21 Project Manager provided status 
updates on cost, schedule, technical performance and risks to executive management for 
oversight via established weekly reports to the Acquisition Director; Monthly Acquisition 
Reports to the Commandant; Quarterly Acquisition Reports to Congress (as approved by 
DHS and OMB); annual reports to Congress and Business Case Exhibit 300 to the Office 
of Management and Budget (OMB) as well as periodic milestone decision reviews to the 
CGARC and DHS Joint Requirements Council (JRC)/Investment Review Board (IRB). 

Page 28 of the GAO briefing states that "executive management did not take action in 
response to the risks presented" at the DHS IRB in Jun 2005. To the contrary, the Top 
Ten project risks were presented to the CGARC in Apr 2005 in preparation for the DHS 
IRB. The Vice Commandant (in his role as the Coast Guard Acquisition Executive) led 
the discussion regarding ongoing mitigation actions for the risks identified. A number of 
these risks were successfully mitigated and closed, and the remaining risks were 
highlighted in the DHS JRC/IRB investment review process. The DHS Deputy Secretary 
specifically addressed the cost, environmental, real estate and vessel subsystem risks in 
his Acquisition Decision Memorandum of 23 Jun 2005. 

The Department greatly appreciates the efforts of the GAO audit team and acknowledges 
that the scope of the evaluation was broad, requiring the review and analysis of a myriad 
of project plans and documentation within an extremely compressed timeline to meet 
Congressional deadlines. The Coast Guard will continue to aggressively pursue the 
recommendations of the GAO report in order to ensure RESCUE 21 provides critical life- 
saving capabilities, and that the acquisition itself is effectively managed. 

Thank you again for the opportunity to comment on this draft report and we look forward 
to working with you on future homeland security issues. 

Sincerely, 

Steven J. Pecinovsky 
Director 

Departmental GAO/OIG Liaison Office 
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